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ABSTRACT 


Measuring  the  effectiveness  of  information  systems  (IS) 
is  an  issue  that  has  generated  debate  and  research  among 
academics  and  practitioners.  This  thesis  consolidates  the 
numerous  and  various  approaches  to  measuring  IS  effectiveness 
into  six  general  schools  of  thought:  user  satisfaction,  system 
usage,  performance/  usefulness,  productivity,  value  analysis 
and  cost-benefit  analysis.  It  then  presents  a  model  for 
examining  the  various  linkages  that  exist  among  the  IS 
effectiveness  measures.  These  linkages  include:  user 
satisfaction  and  system  usage,  system  usage  and  performance, 
performance  and  productivity,  and  productivity  and  cost 
justification . 

This  research  provides  a  user  a  summary  of  the  IS 
effectiveness  literature  of  the  past  two  decades  and  a 
consolidated  reference  for  measuring  the  effectiveness  of 
information  systems. 
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I.  INTRODUCTION 


A.  THE  PROBLEM  AREA 

An  information  system  (IS)  derives  its  value  from  the 
effect  it  has  on  the  performance  of  an  organization  (Coleman 
and  Riley,  1972)  .  This  is  an  issue  for  IS  researchers  and 
practitioners  (Alavi,  1989) .  Researching  this  issue,  three 
questions  are  considered: 

1.  What  is  IS  effectiveness? 

2.  What  evaluation  techniques  are  used  to  measure  IS 
effectiveness? 

3.  What  relationships  exist 

a)  among  evaluation  techniques,  and 

b)  between  evaluation  techniques  and  other  factors? 

To  answer  these  questions,  relevant  literature  in  various 
IS  journals  (Appendix  A)  published  between  1970  and  1991  was 
reviewed.  This  time-frame  was  chosen  because  it  reflects  the 
period  when  researchers  addressed  effectiveness  measures  of 
computer  information  systems.  These  systems  revolutionized  the 
way  of  doing  business  in  organizations  and  it  became  necessary 
to  explore  the  benefits  of  IS  investments.  This  thesis 
integrates  the  IS  effectiveness  literature  and  provides  a 
consolidated  reference  for  measuring  the  effectiveness  of 
information  systems. 
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B.  DEFINING  IS  EFFECTIVENESS 

The  definition  of  effectiveness  in  IS  has,  itself,  been  a 
topic  of  research  and  discussion  (Carlson,  1974;  Radecki, 
1976;  King  and  Schrems,  1978;  Ginzberg,  1978;  Kleijnen,  1979; 
Chandler,  1982;  Markus  and  Robey,  1983;  Maggiolini,  1986; 
Dominick,  1987;  Trice  and  Treacy,  1988;  Hufnagel,  1990; 
Amoroso,  1990;  Haga  and  Zviran,  1990)  .  Researchers  are  in 
agreement  that  IS  effectiveness  focuses  on  the  organizational 
effects  produced  by  an  information  system  (Haga  and  Zviran, 
1990)  .  These  effects  include:  user  satisfaction  with  an 
information  system  (Lucas,  1974;  King  and  Rodriguez,  1978; 
Robey  and  Zeller,  1978;  Mahmood  and  Becker,  1985),  usage  of  an 
information  system  (Lucas,  1975;  Ginzberg,  1978;  Trice  and 
Treacy,  1988),  productivity  of  an  information  system 
(Maggiolini,  1986) ,  performance  and  usefulness  of  an 
information  system  (Franz  and  Robey,  1986;  Miller  and  Doyle, 
1987),  cost-benefit  analysis  (Keim  and  Janaro,  1982;  Lay, 
1985),  and  value-analysis. 

These  various  meanings  of  effectiveness  can  also  be  linked 
to  provide  additional  insights  into  IS  effectiveness.  Such 
linkages  include;  how  user  satisfaction  and  system  usage 
affect  one  another  (Fishbein  and  Ajzen,  1975;  Srinivasan, 
1985;  Mahmood  and  Becker,  1985;  Baroudi,  et .  al,  1986),  the 
effect  of  an  information  system's  performance  on  productivity 
(Bearman,  et .  al,  1985;  Schwartz,  1986),  how  productivity 
relates  to  cost-benefit  analysis  (Schwartz,  1986),  and  the 
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impact  of  user  involvement  on  system  usage  and  user 
satisfaction  (Baroudi,  et .  al,  1986;  Baronas  and  Louis,  1988). 

The  problem  for  a  user  is  deciding  which  IS  effectiveness 
measure  to  apply  to  a  given  system.  In  Chapter  II,  individual 
measures  of  IS  effectiveness  are  discussed  and  application 
examples  are  presented.  First,  it  is  appropriate  to  address 
the  initial  question  posed  in  this  study:  "What  is  IS 
effectiveness?" 

According  to  Carlson  (1974),  IS  effectiveness  is  concerned 
with  those  effects  on  an  organization  which  result  from  the 
development  and  use  of  an  information  system.  The  following 
are  some  examples  of  the  effects  Carlson  describes: 


1.  Information  systems  will  reduce  or  increase  the 
cost  of  operations. 

2.  Information  systems  will  reduce  or  increase  the 
growth  rate  in  employment. 

3.  Information  systems  will  reduce  or  increase 

clerical  work. 

4.  Information  systems  will  improve  reporting  by 

providing  more  accurate  and  mc’'e  timely  reports, 
with  less  effort. 

5.  Information  systems  will  improve  or  reduce 

productivity . 

6.  Information  systems  will  improve  decision-making, 
by  providing  more  timely  and  more  accurate 
information,  by  stimulating  more  interaction  among 
decision-makers,  and  by  providing  better 
projections  of  the  effects  of  decisions. 

7.  Information  systems  will  alter  the  attitudes, 

activities  and  interactions  of  administrators. 
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Hamilton  and  Chervany  (1981)  de ' me  IS  effectiveness  as 
the  "accomplishment  of  objectives."  They  assert  two  general 
views  concerning  what  IS  effectiveness  means  and  how  it  should 
be  measured.  The  first  view,  the  goal-centered  viewstates  that 
the  way  to  assess  information  systems  effectiveness  is  first 
to  determine  the  task  objectives  of  the  system  and  then  to  ^ 

develop  criterion  measures  to  assess  how  well  the  objectives 
are  being  achieved.  Effectiveness  is  determined  by  comparing 
performance  to  objectives.  An  example  of  the  goal-centered 
view  of  IS  effectiveness  xS  to  compare  actual  costs  and 
benefits  to  budgeted  costs  and  benefits. 

Second  is  the  system-resource  view.  Effectiveness  is 
determined  by  attainment  of  a  normative  state,  e.g.,  standards 
for  "good"  practices.  In  this  view,  effectiveness  is 
conceptualized  in  terms  of  resource  viability  rather  than  in 
terms  of  specific  task  objectives.  For  example,  IS 
effectiveness  in  terms  of  human  resources  might  be  indicated 
by  the  nature  of  communication  and  conflict  between  IS  and 
user  personnel,  user  participation  in  system  development,  or 
user  job  satisfaction. 

Other  definitions  of  IS  effectiveness  are  embedded  in  the 
evaluation  techniques.  For  example,  proponents  of  productivity 
as  a  measure  of  IS  effectiveness  do  not  define  effectiveness  « 

explicitly  and  separately  from  productivity.  Rather,  they 
demonstrate  IS  effectiveness  (or  lack  thereof)  through  /arious 
productivity  measures.  In  the  following  chapter  on  measures  of 
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IS  effectiveness,  other  definitions,  implicit  to 
effectiveness  me^iSure,  will  emerge. 


% 


the 
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II.  MEASURES  OF  INFORMATION  SYSTEMS  EFFECTIVENESS 


A.  GENERAL 

Measurement  of  the  effectiveness  of  informat-ion  systems  is 
an  issue  that  has  generated  debate  and  research  among 
academics  and  practitioners.  Approaches  that  have  been 
advocated  include  usage  estimation,  user  satisfaction, 
incremental  performance  in  decision  making  effectiveness, 
cost-benefit  analysis,  information  economics,  utility 
analysis,  performance  factors,  usefulness  of  information 
output,  productivity,  value-analysis,  cost-savings,  and 
benefits  produced  by  a  system  related  to  costs. 

To  consolidate  these  numerous  and  varied  approaches  to 
measuring  IS  effectiveness,  the  ones  that  were  most  prevalent 
in  the  IS  literature  were  extracted  and  classified  into  six 
general  schools  of  thought: 

1.  User  Satisfaction 

2.  System  Usage 

3.  Performance/  Usefulness 

4.  Productivity 

5.  Cost-Benefit  Analysis 

6.  Value-analysis 
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Each  school  of  thought  is  presented,  individually,  in 
terms  of  what  the  IS  ef  fe<^tiveness  measure  is,  why  it  is 
important  and  valid,  and  its  application. 

B.  USER  SATISFACTION 

User  satisfaction  with  information  systems  is  important 
because  of  its  potential  effects  on  organizational  goals, 
quality  of  user  work  life,  and  extent  of  voluntary  usage  of 
systems  (Galletta  and  Lederer,  1989) .  Researchers  have  argued 
that  use  of  a  system  introduced  by  an  IS  organization  is 
directly  related  to  a  user  community's  sense  of  satisfaction 
(Mahmood  and  Becker,  1985) .  Reliable  measurement  of  user 
satisfaction  is  important  for  providing  a  summative  evaluation 
for  the  researcher  and  a  formative  evaluation  for  the 
practioner  (Galletta  and  Lederer, 1989) .  A  researcher  might 
want  to  investigate  relationships  of  user  satisfaction  with 
other  variables.  Thus,  user  satisfaction  might  predict 
outcomes  or  be  predicted  by  other  variables  such  as  training. 
Once  established,  such  relationships  may  provide  better 
understanding  of  the  IS  environment. 

On  the  other  hand,  a  practitioner  might  be  more  interested 
in  user  satisfaction  as  a  feedback  mechanism  to  uncover  user 
perception  of  strengths  and  weaknesses.  Management  could  use 
the  strengths  for  reinforcement  and  recognition  and  the 
weaknesses  as  the  basis  for  making  improvements.  Cerullo 
(1980)  ,  in  his  research,  emphasized  the  importance  of  user 
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satisfaction  by  revealing  that  IS  professionals  consider  user 
attitudes  to  be  the  single  most  important  success  factor. 

While  seeking  a  model  of  computer  user  satisfaction, 
Bailey  and  Pearson  (1983)  defined  user  satisfaction  as  the  sum 
of  the  user's  weighted  reactions  to  a  set  of  factors. 

Si:  =  ^  RirWi: 

where  R..  =  The  reaction  to  factor  j  by  individual  i. 

W,.  =  The  importance  of  factor  j  to  individual  i. 

The  model  suggests  that  satisfaction  is  the  sum  of  one's 
positive  and  negative  reactions  to  a  set  of  factors.  An 
individual's  feeling  must,  in  this  model,  be  placed  somewhere 
between  a  "most  negative"  reaction  and  a  "most  positive" 
reaction.  Implementation  of  the  model  centers  on  two  different 
requirements.  First,  the  set  of  factors  comprising  the  domain 
of  satisfaction  must  be  identified.  Second,  a  vehicle  for 
scaling  an  individual's  reaction  to  those  factors  must  be 
found. 

Bailey  and  Pearson  (1983)  compiled  a  list  of  39  factors 
making  up  the  domain  of  user  satisfaction.  These  factors  and 
their  definitions  appear  in  Appendix  B.  In  an  empirical  test, 
the  list  of  factors  was  compared  to  interview  responses  from 
32  middle  manager  users  in  eight  different  organizations.  The 
respondents  ranked  factors  in  order  of  relative  importance  to 
their  own  satisfaction.  The  ranking  suggested  that  individuals 
differ  in  the  factors  which  affect  their  perception  of 
satisfaction.  The  dimensionality  and  intensity  of  an 
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individual's  reaction  to  a  particular  factor  was  measured 
using  the  seven-interval  scale  shown  in  Figure  2.1. 


This  vehicle  for  measuring  a  user's  reactions  hinged  on 
the  definition  of  satisfaction  as  a  bi-dimensional  attitude 
affected  by  a  variety  of  factors  (Bailey  and  Pearson,  1983) . 

Ives,  et .  al.(1983)  refined  and  abbreviated  Bailey  and 
Pearson's  (1983)  instrument  into  a  short,  13-itt;m 
questionnaire  (Appendix  C)  ,  referred  to  as  the  "detailed" 
instrument  (Galletta  and  Lederer,  1989) .  Factor  analysis 
identified  three  factors  of  user  satisfaction:  the  information 
system  product  (quality  of  output) ,  support  (quality  and 
service  of  the  IS  function),  and  involvement  (knowledge  of  the 

» 

systems  or  involvement  in  the  development  of  the  systems) .  The 
*  detailed  user  satisfaction  instrument  is  increasingly  used  in 

IS  attitude  research  and  according  to  Galletta  and  Lederer 
(1989),  is  probably  the  most  refined  measure  of  user 
satisfaction  to  date.  Additionally,  Sanders  (1984)  developed 
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an  instrument  to  measure  satisfaction  with  decision  support 
systems . 

There  are  criticisms  of  the  user  satisfaction  surveys. 
These  instruments  attempt  to  measure  the  success  of  a  system 
in  terms  of  users'  feelings  -  are  they  satisfied  or 
dissatisfied  with  an  information  system  (Hufnagel,  1990)  . 
Proponents  of  the  user  satisfaction  approach  argue  that  the 
subjective  assessments  that  result  provide  "a  meaningful 
surrogate  for  the  critical  but  unmeasurable  result  of  an 
information  system,  namely,  changes  in  organizational 
effectiveness"  (Ives,  et .al . , 1983) .  However,  the  subjective 
aspect  of  user  satisfaction  surveys  has  led  some  researchers 
to  question  the  widespread  acceptance  of  these  evaluation 
tools.  Davis  and  Srinivasan  (1988)  point  out  the  user 
satisfaction  approach  ultimately  hinges  on  three  assumptions: 
first,  the  perception  of  the  user  with  regard  to  the  system 
being  used  is  an  accurate  indicator  of  system  effectiveness; 
second,  the  perceptions  of  several  users  of  a  system  can  be 
aggregated  to  arrive  at  an  overall  assessment  of  the  system 
under  study;  and  third,  user  satisfaction  with  a  system  can  be 
accurately  measured. 

An  automated  system  which  is  perceived  by  the  user  to  be 
unsuccessful  is  unsuccessful,  regardless  of  how  good  it  may 
be,  and  vice  versa  (Martin  and  Trumbly,  1986) .  Hufnagel  (1990) 
concluded  in  his  empirical  research  that  users  discounted  the 
contribution  of  an  IS  when  things  went  well  and  blamed  the 
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system  when  things  went  poorly.  This,  he  claimed,  suggested 
that  user  satisfaction  may  be  a  "less  than  adequate”  surrogate 
measure  for  system  effectiveness  because  the  actual 
contribution  of  an  IS  is  ambiguous  or  difficult  to  quantify 
from  a  user's  perspective.  For  example,  it  may  be  troublesome 
as  a  surrogate  for  effectiveness  when  users  are  inexperienced 
at  performing  the  task  in  question,  do  not  have  a  good 
understanding  of  how  the  system  actually  works,  or  are 
otherwise  unable  to  judge  the  impact  system  use  has  had  on 
their  outcomes.  Expert  systems  and  some  types  of  decision 
support  systems  that  are  specifically  designed  to  aid  novice 
users  may  be  especially  vulnerable  to  the  problem  of  causal 
attributions  if  users  expect  that  the  answers  provided  by  the 
system  will  necessarily  be  correct  (Hufnagel, 1990) . 

With  its  shortcomings,  academicians  and  practitioners 
agree  that  the  IS  field  has  advanced  as  a  result  of  the 
research  on  user  satisfaction.  The  development  of  instruments 
with  which  to  measure  user  satisfaction  has  encouraged  more 
widespread  incorporation  of  this  approach  in  research  and  its 
use  by  practitioners  in  evaluating  system  effectiveness.  To 
the  extent  that  the  scales  of  Bailey  and  Pearson  (1983) ,  Ives, 
et .  al,,  (1983)  and  others  are  valid  and  reliable,  they 
provide  information  about  the  overall  satisfaction  with  IS 
products  and  services.  In  addition,  they  provide  a  standard 
for  making  comparisons  across  organizational  units  and  over 
time  within  units.  They  are  also  relatively  simple  and 


inexpensive  to  administer.  Without  minimizing  these 
contributions,  a  number  of  theoretical  and  practical  issues 
related  to  user  satisfaction  remain  to  be  resolved  (Melons, 
1990)  . 

C .  SYSTEM  USAGE 

In  addition  to  user  satisfaction,  an  approach  to 
evaluating  IS  effectiveness  that  is  prevalent  in  the  IS 
literature  is  behavioral  and  focuses  on  system  usage.  This 
approach  was  proposed  by  Ein-Dor  and  Segev  (1978)  and  has 
often  been  used  in  empirical  research  (Cheney  and  Dickson, 
1982;  Baroudi,  et ,  al .  ,  1986).  System  usage  can  be  subdivided 
into  offline  usage,  where  user-system  interaction  is  limited 
to  the  use  of  printed  reports  output  by  the  system  or  to 
accesses  through  an  intermediary  and  online  usage,  where  a 
user  interacts  with  a  system  through  a  terminal. 

The  amount  of  use  an  individual,  group,  or  organization 
makes  of  an  information  system  is  a  variable  in  measuring  IS 
effectiveness.  For  example,  poor  systems  may  require  more 
usage  and  vice  versa.  Many  times  it  is  used  as  an  independent 
variable  when  studying  the  impact  that  an  information  system 
has  had  on  the  process,  structure  or  performance  of  an 
organization  (Trice  and  Treacy,  1988) .  System  use  or 
utilization  is  implicitly  defined  by  researchers  as  either  the 
amount  of  effort  expended  interacting  with  an  information 
system  or,  less  frequently,  as  the  number  of  reports  or  other 
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information  products  generated  by  the  information  system  per 
unit  time  (Trice  and  Treacy,  1988) .  Examples  include  frequency 
and  the  number  of  computer  sessions,  connect  time,  time  spent 
using  different  system  functions,  number  of  records  updated, 
and  keystrokes  or  carriage  returns.  There  are  instances, 
however,  where  a  high  level  of  system  usage  is  not  a  sign  of 
successful  implementation  or  IS  effectiveness.  Two  different 
types  of  use  should  be  distinguished:  voluntary  and 
involuntary.  According  to  Lucas  (1978),  the  use  of  a  model  is 
a  good  indicator  when  usage  is  voluntary.  If  usage  is 
voluntary,  then  a  high  level  of  use  means  that  the  user 
perceives  some  benefits  from  the  information  system. 

Although  system  usage  is  a  variable  that  appears  in 
various  aspects  of  measuring  IS  effectiveness,  researchers 
find  it  inadequate  as  a  stand-alone  measure  (Ginzberg,  1978) . 
Linked  with  performance  and  user  satisfaction,  system  usage 
has  broader  application.  These  relationships  are  discussed  in 
Chapter  III  which  addresses  linkages  among  effectiveness 
measures . 

D .  SYSTEM  PERFOBMANCE/USEFULMESS 

The  performance/usefulness  of  information  systems  can  be 
evaluated  from  two  different  perspectives.  One  focuses  on  the 
computer  system  domain  and  the  other  on  user  domain.  Each  has 
its  own  goals  and  measures.  In  the  computer  system  domain, 
performance  is  measured  in  terms  of  resource  utilization,  cost 
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and  efficiency  while  in  the  user  domain  throughput, 
reliability,  and  response  times  are  common  measures. 

In  some  operating  environments,  it  is  quite  possible  to 
design  a  system  such  that  good  performance  for  one  or  more 
users  is  gained  at  the  expense  of  other  users.  Furthermore, 
because  system  resources  may  be  used  by  different  users, 
improving  the  performance  characteristics  of  one  or  more 
resources  for  the  benefit  of  specific  users  may  have  a 
detrimental  effect  on  overall  performance.  The  problem 
presented  to  the  analyst  is  to  configure  a  system  which 
satisfies  the  users'  effectiveness  criteria  while 
simultaneously  achieving  system  performance  criteria 
(Chandler,  1982) . 

Generally,  system  performance  can  be  thought  of  as  "is  the 
system  doing  what  it  is  supposed  to  do?"  When  the  system  has 
a  high  percentage  of  "up-time, "  produces  complete  and  accurate 
information  in  a  timely  fashion,  then  performance  is  thought 
to  be  favorable.  Procedures  and  tools  should  be  developed  and 
implemented  to  monitor  resource  performance  against 
performance  standards  and  targeted  objectives  (Ameen,  1989) . 
Substandard  performance  may  portend  inefficiency  or 
ineffectiveness  in  providing  a  particular  user  service.  When 
evaluating  performance,  Ameen  (1989)  proposed  the  following 
four  steps: 

1.  Identify  critical  performance  variables 
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2.  Establish  methods  to  collect,  measure  and  analyze 
performance 

3.  Determine  the  nature  of  corrective  action 

4.  Evaluate  the  evaluation  process  itself 

One  of  the  most  difficult  tasks  is  selection  of 
performance  criteria.  While  common  criteria  such  as  hardware 
or  software  response  time  are  typically  used  throughout  the 
industry,  it  is  critical  to  select  the  variables  which  best 
indicate  resource  performance  for  the  particular  organization 
of  interest.  There  are  two  general  categories  of  measures: 
variables  pertaining  to  efficiency  and  variables  pertaining  to 
effectiveness  (Ameen, 1989) .  If  efficiency  measures  are 
emphasized  over  effectiveness  measures,  costs  may  decrease  and 
usage  and  throughput  may  increase,  but  the  quality  and 
timeliness  of  the  resoure  output  will  probably  decline.  The 
reverse  is  true  if  effectiveness  scores  are  weighted  more 
heavily  than  efficiency  measures. 

Other  techniques  for  evaluating  performance  include 
benchmarking  (the  sizing  of  a  targeted  CPU  against  the 
workload)  or  stress  testing  (the  measurement  of  workload 
impact  of  concurrent  usage)  and  simulation  (the  prediction  of 
workload  behavior  under  varying  conditions) .  A  newer  technique 
is  the  expert  performance  control  monitoring  system  which  uses 
artificial  intelligence  to  direct  users  through  a  series  of 
questions  and  to  recommend  courses  of  action.  Hardware  and 
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software  monitors  measure  performance  of  computer  equipment 
and  applications.  These  tools  assist  in  understanding  system 
behavior,  identifying  bottlenecks  and  determining  the  need  for 
system  tuning  and  capacity  planning. 

Realizing  that  measures  of  system  effectiveness  are 
interrelated,  the  major  purpose  of  performance  evaluations  are 
to  create  and  maintain  good  user  relations;  to  enable 
personnel  to  make  decisions  about  the  computing  environment; 
and  to  ensure  that  the  needed  service  is  being  provided  to  the 
organization  in  an  efficient  and  effective  manner  (Ameen, 
1989)  . 

E.  PRODUCTIVITY 

Productivity  is  another  approach  to  measuring  IS 
effectiveness.  In  the  area  of  computer  use  and  productivity. 
Mason  (1984)  suggests  that  information  systems  generally  serve 
three  purposes.  One  of  them  is  reducing  costs;  another  is 
increasing  revenues;  the  third  is  prcviding  better  services. 
He  suggests  that  the  main  benefits  of  IS  are  the  savings  which 
are  possible  in  work-time  when  the  systems  are  implemented. 
Maggiolini  (1986)  advocates  that  IS,  particularly  in  the  area 
of  office  automation,  improves  efficiency  by  automating  all  or 
some  parts  of  the  office  information  process;  eliminating  some 
of  the  transformations  of  medium;  eliminating  or  reducing  the 
shadow  activities  or  functions;  and  speeding  up  the 
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information  process  itself.  This,  he  claims,  increases  overall 
product ivity . 

In  defining  productivity,  Bain  (1982)  contends  that 
productivity  is  not  a  measure  of  output  produced.  He  says 
instead  that  it  is  a  measure  of  how  well  resources  are 
combined  to  accomplish  specific  results.  He  explains  that  a 
concept  of  productivity  must  account  for  an  interplay  between 
factors  such  as  quality,  availability  of  materials,  scale  of 
operations,  the  rate  of  capacity,  the  rate  of  capacity 
utilization,  the  attitude  and  skill  level  of  the  work  force, 
and  the  motivation  and  effectiveness  of  managemient .  The  way  in 
which  these  factors  interrelate  has  an  important  bearing  on 
the  resulting  productivity. 

Productivity  is  the  ratio  of  the  number  of  units  of  output 
to  the  number  of  units  of  input; 

Productivity  =  output /input . 

Productivity  rises  when  the  number  of  units  of  output 
increases  while  input  holds  steady;  or  conversely,  when  the 
number  of  units  of  output  remains  fixed  and  the  number  of 
units  of  input  is  decreased.  Productivity  can  be  measured  by 
converting  input  and  output  units  to  their  respective  dollar 
equivalents . 

Borko  (1983)  says  the  concept  of  productivity  measurement 
is  clear,  and  the  formula  relatively  easy  to  apply  in 
situations  where  the  input  and  output  units  are  readily 
identifiable  and  countable.  He  contends,  however,  that  such 
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measures  are  much  more  difficult  to  apply  when  measuring  the 
productivity  of  office  workers,  and  especially  knowledge 
workers.  The  output  of  an  office  is  varied  and  largely 
intangible  (Borko,  1983) . 

He  states  that  effective  management  is  largely  dependent 
upon  the  availability  of  current  and  reliable  information 
processed  to  facilitate  decision  making  and  enhance 
productivity.  He  says  that  the  studies  that  have  been 
performed  on  office  information  processing  have  concerned 
clerical  productivity  which  lends  itself  more  readily  to 
quantitative  measure.  He  further  contends  that  clerical 
workers,  however,  account  for  only  one-fourth  to  one-third  of 
all  white  collar  office  costs,  and  the  clerical  worker  alone 
is  not  the  greatest  source  of  potential  savings.  The  larger 
segment  of  the  office  environment  includes  knowledge  workers, 
e.g.  managers,  professionals  and  technical  workers.  Since  the 
cost  of  knowledge  workers  is  significantly  greater  than  the 
cost  of  clerical  personnel,  an  increase  in  knowledge  worker 
productivity  can  result  in  significant  savings  (Borko,  1983) . 
Since  knowledge  workers  exert  the  greatest  influence  on  the 
decision  making  process,  changes  in  their  IS  practices  could 
have  a  major  impact  on  their  productivity  and  the  productivity 
of  other  office  workers- 

Sink  defines  productivity  measurement  as  "the  selection  of 
physical,  temporal,  and  perceptual  measures  for  both  input 
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variables  and  output  variables  and  the  development  of  a  ratio 
of  output  measure  (s)  to  input  measure (s)."  (Sink,  1985,  p.  25) 

Sink  says  there  are  two  basic  categories  of  pure 
productivity  measures.  The  first  are  static  productivity 
ratios  in  which  measures  of  output  are  divided  by  measures  of 
input  for  a  given  period  of  time.  The  second  category  are 
dynamic  productivity  indexes  which  give  a  static  productivity 
ratio  at  some  previous  peri ' d  in  time.  There  are  three  types 
of  productivity  measures  within  each  category: 

1.  The  partial  factor  measure  which  uses  one  class  of 
input  such  as  labor  or  capital, 

2.  the  multifactor  measure  which  uses  more  than  one 
class  and, 

3.  the  total  measure  which  uses  all  classes  of 
inputs . 

Each  of  the  three  types  represents  a  ratio  of  output  to 
input.  However,  they  differ  in  terms  of  how  much  input  is 
captured  in  the  denominator  of  the  equation. 

Sink  defined  productivity  as  the  relationship  between 
quantities  of  outputs  from  a  system  and  quantities  of  inputs 
into  that  same  system.  Dissecting  this  definition,  it  can  be 
seen  that  the  numerator  contains  an  aspect  of  effectiveness  in 
the  way  of  quality  and  quantity.  While  on  the  other  hand,  the 
denr>minator  contains  an  aspect  of  efficiency  in  tne  way 
resources  are  actually  consumed. 


19 


Mason  (1978)  says  defining  and  measuring  the  output  of  any 
system  is  difficult,  especially  if  the  output  is  rather 
intangible.  Much  of  the  original  development  of  output 
measures  and  productivity  analysis  occurred  in  industry  where 
such  tangible  measures  such  as  "automobiles  produced  per  man¬ 
hour"  or  "ton  miles  moved  per  man-hour"  are  reasonable 
output/input  indicators  of  productivity.  He  describes  an 
im.portant  class  of  systems  for  which  output  measures  are 
difficult  to  define.  These  are  systems  whose  output  is 
primarily  information.  Essentially,  information  involves  the 
means  by  which  one  system  affects  another.  So  an  information 
system  is  a  component  of  any  system  and  a  critical  element  in 
the  linkage  between  many  diverse  systems  (Mason, 1978) .  Systems 
which  produce  information  as  an  output  include  accounting, 
data  processing,  research  and  development,  education, 
communication  media,  and  entertainment.  These  systems  not  only 
produce  information  as  an  output  but  also  consume  resources  in 
the  process.  Consequently,  concerns  for  efficiency  require 
that  the  output  of  these  systems  be  related  to  the  input 
resources  consumed.  Only  then  can  managerial  judgements  be 
made  as  to  the  appropriateness  of  the  allocations.  This 
measurement  requires  understanding  the  fundamental  nature  of 
information  and  of  alternative  ways  of  measuring  it. 

Shannon  and  Weaver  (1963)  distinguish  among  the  technical 
level,  semantic  level  and  influence  level  when  measuring  an 
information  system's  output.  The  technical  level  refers  to  the 
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physical  manipulation  of  data  and  media.  The  semantic  level 
refers  to  the  logical  units  of  meaning  handled  by  the  system. 
Finally,  the  influence  level  refers  to  units  of  change  in 
behavior,  attitude  or  cognition  created  by  the  system. 

Shannon  and  Weaver  (1963)  explain  that  basic  productivity 
measurement  may  be  obtained  at  the  technical  level  by 
comparing  the  number  of  units  of  output  as  measured  by  the 
units  to  a  measure  of  resources  consumed  by  the  system;  for 
example,  bits  processed  per  dollar  of  expenditure.  Basic 
productivity  measures  are  obtained  at  the  semantic  level  by 
relating  the  number  of  units  of  output  to  a  measure  of 
resources  consumed  by  the  system;  for  example  words  processed 
per  dollar  of  expenditure.  Measures  of  productivity  calculated 
at  the  semantic  level  will  be  different  from  measures  of 
productivity  calculated  at  the  technical  level  and  are  not 
necessarily  correlated. 

Finally,  he  says,  the  measure  of  output  at  the  influence 
level  is  the  number  of  units  of  change  or  affect  that  takes 
place  in  the  receiver  as  a  result  of  receiving  information 
from  the  producer.  A  problem  to  be  solved  in  obtaining 
measures  of  output  at  the  influence  level  is  to  determine  the 
amount  of  the  response  or  change  due  to  the  information 
received  from  the  producer  and  not  due  to  other  causes 
(Shannon  and  Weaver,  1963)  . 

In  a  given  circumstance,  one  would  want  to  choose  one  or 
more  of  these  productivity  measures  to  describe  the  key 
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aspects  of  the  system  under  study.  These  output  measures  may 
be  related  to  a  variety  of  input  measures  to  form  productivity 
measures  for  the  information  system  as  a  whole. 

While  productivity  is  an  applicable  measure  for  the 
effectiveness  of  many  information  systems,  it  is  less  than 
adequate  for  assessing  decision  support  systems  and  the  like. 
In  that  case,  the  value-analysis  approach  would  be  more 
applicable.  It  is  discussed  in  the  following  section. 

F .  VALUE  ANALYSIS 

Value  analysis  is  an  approach  used  for  measuring  the 
effectiveness  of  decision  support  systems  (DSS) .  Once  a  DSS 
has  been  up  and  running  in  an  organization,  the  question  may 

exist  whether  to  update  or  scrap  the  DSS.  Value  Analysis 

0 

allows  the  evaluator  to  determine  what  the  user  is  willing  to 
pay  in  order  to  keep  his  present  DSS  and  to  determine  if  there 
are  other  benefits  that  can  be  added  to  the  DSS.  This  method 
was  first  introduced  by  Keen  (1981)  and  was  divided  into 
eleven  steps  by  Smith  (1983) .  The  eleven  steps  are: 

1.  Define  the  benefits  to  be  obtained  if  a  prototype 
system  is  developed. 

2.  Determine  the  amount  the  users  are  willing  to  pay 
to  obtain  the  benefits. 

3.  Determine  whether  a  prototype  can  be  implemented 
within  the  "cost  threshold"  established  by  the 
user , 

4.  Design  the  prototype  and  measure  its  use  and  cost. 
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5.  Review  and  extend  benefits  list  if  necessary. 

6.  Define  computer  hardware  and  software  which  would 
permit  a  more  complete  system  to  be  evolved  from 
the  prototype. 

7.  Determine  cost  of  expanding  the  system. 

8.  Ask  users  to  determine  level  of  benefits  which 
must  be  obtained  to  justify  investing  in  an 
expanded  system. 

9.  Assuming  users  provide  a  feasible  new  "cost 
threshold,"  design  a  second  version  of  the  system. 

10.  After  the  second  version  is  implemented,  measure 
its  use  and  costs  and  determine  new  cost  threshold 
for  a  possible  third  stage  of  development. 

11.  Continue  steps  5-10  until  users  and  systems 
designers  are  satisfied  that  the  best  solution  has 
been  achieved  under  existing  constraints. 


This  method  allows  the  users  to  determine  what  benefits 
are  important  and  place  a  monetary  figure  to  each  benefit.  The 
monetary  values  are  based  on  what  the  user  will  be  willing  to 
pay  to  obtain  a  specific  benefit.  This  approach  uses  the 
"quick  hit"  approach  when  developin  a  DSS  and  utilizes 
prototypes  to  get  a  working  model  in  a  users'  hands  as  quickly 
as  possible  and  at  a  low  initial  cost. 

The  evolving  DSS  that  Value  Analysis  provides  is  an 
excellent  way  of  continually  evaluating  an  organization's  DSS. 
If  users  are  pleased  with  the  initial  versions  of  the  DSS,  the 
organization  can  continually  update  the  DSS  to  include  more 
functions , 
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6.  COST  BENEFIT  ANALYSIS 


Cost  Benefit  Analysis  (CBA)  is  an  investment  decision 
technique  that  is  applied  to  information  systems.  In  the  early 
stages  of  a  complex  IS  design,  the  lack  of  specific 
requirements,  the  uncertainty  of  needed  manpower  requirements, 
and  the  inability  to  estimate  intangibles  results  in  poor 
attempts  at  cost-benefit  analyses  (Keim  and  Janaro,  1982) . 
Some  researchers  argue  that  there  are  instances  where 
incorrect  IS  decisions  have  been  taken  through  excessive 
reliance  by  the  decision-makers  on  CBA  results  (Lay,  1985) . 
Opinions  on  the  applicability  of  CBA  to  IS  range  from  that  of 
advocating  profit  and  loss  evaluations  to  those  that  would  not 
attempt  any  cost-benefit  analyses  (Mathusz,  1977) .  Normally 
applied  in  the  pre-implementation  phase,  CBA  as  a  planning 
tool  or  justification  in  the  IS  environment  has  its 
shortcomings . 

Cost-benefit  analysis  relies  on  the  fact  that  costs  and 
benefits  can  be  estimated  with  reasonable  accuracy.  If  they 
are  not,  the  net  worth  of  a  system  will  be  incorrect  and 
invalid  decisions  will  be  made,  such  as: 

1 .  Setting  the  incorrect  priority  for  development 

2.  Rejecting  a  potentially  profitable  system 

3.  Accepting  a  non-viable  project  (Lay, 1985). 

Lay  (1985)  states  that  the  primary  purpose  of  CBA  is  to 
measure  the  econmic  return  of  an  information  system  which  will 
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assist  management  in  approving  a  particular  information  system 
for  development.  He  describes  it  as  typically  consisting  of 
the  following  five  steps: 

1.  Defining  the  scope  of  the  project. 

2.  Evaluating  the  direct  and  secondary  costs  and 
benefits  of  the  project. 

3.  Defining  the  life  of  the  project. 

4.  Discounting  the  net  present  values  of  benefits. 

5.  Performing  sensitivity  analyses. 

An  examination  of  the  relevance  of  the  CBA  technique  to  IS 
planning  reveals  that  its  shortcomings  fall  into  two 
categories.  The  first  relates  to  problems  of  estimating  the 
costs  and  benefits  of  IS  projects.  The  second  category  deals 
with  costs  that  cannot  be  measured  at  all  but  should  be 
included  in  the  model. 

Keim  and  Janaro  (1982)  proposed  a  phased  approach  to  cost- 
benefit  analysis.  It  takes  into  account  the  system  design  life 
cycle  and  applies  a  flexible  and  evolutionary  cost-benefit 
analysis  throughout  the  life  cycle.  As  each  stage  in  the 
system  design  defines  a  more  specific  system  configuaration, 
the  updated  CBA  is  completed  using  increasingly  specific  and 
quantifiable  information  and  evaluation.  Such  a  phased 
approach  will  avoid  the  traditional,  unrealistic  and 
inaccurate  CBA  that  rely  on  a  single  application  at  a  very 
early  stage  of  the  information  system  design. 
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H.  SUMMARY 


From  the  numerous  and  varied  approaches  to  measuring  IS 
effectiveness,  the  ones  most  prevalent  in  the  IS  effectiveness 
literature  were  extracted  and  classified  into  six  general 
schools  of  thought.  Those  classifications  are:  user 
satisfaction,  system  usage,  performance/usefulness, 
productivity,  value  analysis  and  cost  benefit  analysis.  These 
schools  of  thought  for  measuring  effectiveness  have  been 
discussed  individually  in  terms  of  what  they  are,  their 
importance  and  application.  In  the  next  chapter,  linkages 
among  these  six  IS  effectiveness  measures  be  addressed.  This 
discussion  will  be  based  on  a  conceptual  model  presented  at 
the  beginning  of  the  following  chapter. 
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III.  LINKAGES  AMONG  IS  EFFECTIVENESS  MEASURES 


The  IS  effectiveness  literature  of  the  past  two  decades 
contained  reviews  of  various  linkages  which  existed  among  the 
various  IS  effectiveness  measures.  these  linkages  are 
summarized  in  the  model  presented  in  Figure  3.1.  This  model 
depicts  relationships  among  the  six  schools  of  thought  for 
measuring  IS  effectiveness.  In  the  following  sections,  these 
linkages  are  discussed. 


Figure  3.1.  Model  of  Linkages  Among  IS  Effectiveness  Measures 
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A.  USER  SATISFACTION  AND  SYSTEM  USAGE 


The  fundamental  aim  of  an  organizational  information 
system  is  to  improve  individual  decision-making  performance, 
and  ultimately  organizational  effectiveness  (Raymond,  1990). 
The  difficulty  in  empirically  assessing  system  effectiveness 
in  this  way  has  led  researchers  to  adopt  surrogate  constructs 
that  are  more  easily  measurable.  Of  the  two  main  approaches 
for  evaluating  information  system  success,  the  first  one  is 
behavioral  and  focuses  on  system  usage.  This  approach  was  the 
one  proposed  by  Ein-Dor  and  Segev  (1978) .  System  usage  can  be 
subdivided  into  offline  usage,  where  user-system  interaction 
is  limited  to  the  use  of  printed  reports  output  by  the  system 
or  to  accesses  through  an  intermediary,  and  into  online  usage, 
where  the  user  interacts  himself  with  the  system  through  a 
terminal;  these  are  fundamentally  different  types  of  user 
behaviors  that  are  not  necessarily  related  (Srinivasan,  1985) . 

The  second  approach  in  evaluating  success  centers  on  user 
attitudes,  more  specifically  on  user  satisfaction  with  various 
aspects  of  an  information  system.  This  approach  is  now  the 
most  common  in  the  literature  (Mahmood  and  Becker,  1985; 
Raymond,  1985;  Srinivasan  and  Kaiser,  1987;  Tait  and  Vessey, 
1988) .  Results  of  empirical  studies  have,  however,  been  mixed 
in  regard  to  the  relationship  between  system  usage  and  user 
satisfaction.  While  some  researchers  did  not  find  the  two 
types  of  measures  to  be  associated  significantly  (Cheney  and 
Dickson,  1982),  others  have  (Ein-Dor,  Segev,  Blumenthal  and 
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(1986) 


Millet,  1984;  Raymond,  1985).  Baroudi,  et .  al., 
suggested  that  the  use  of  both  approaches  to  measure  system 
effectiveness  is  warranted  in  most  research  situations. 

1 .  User  Satisfaction  Affects  System  Usage 

User  satisfaction  is  an  attitude  toward  the 
information  system  while  system  usage  is  a  behavior.  Fishbein 
and  Ajzen's  (1975)  model  of  attitudes  and  behaviors  suggests 
that  attitudes  toward  an  object  (in  this  case  an  information 
system)  will  influence  intentions  and  ultimately  influence 
behavior  with  respect  to  that  object  (the  use  of  the  system  or 
its  outputs)  .  This  framework  can  be  interpreted  as  supporting 
the  model  (Figure  3.1)  in  that  user  information  satisfaction 
(an  attitude)  will  lead  to  system  usage  (a  behavior) . 
Empirical  studies  have  shown  that  the  user's  satisfaction  with 
the  system  will  lead  to  greater  system  usage  (Baroudi,  et . 
al.,  1986;  Lucas,  1984). 

For  almost  all  information  systems,  some  use  of  the 
system  is  required,  for  example,  it  may  be  the  only  way  to 
process  orders  or  to  compute  the  payroll.  However,  it  is 
expected  that  the  use  of  systems  beyond  basic  input  and 
transactions  processing  is  voluntary.  For  example  a  user 
generally  decides  voluntarily  to  perform  an  extensive  analysis 
of  an  output  report  instead  of  just  glancing  at  it.  Attitudes 
have  an  action  component  and  favorable  attitudes  are 
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consistent  with  high  levels  of  voluntary  use  of  information 
systems  (Lucas,  1974) . 

2 .  System  Usage  Affects  User  Satisfaction 

Dissonance  theory  (Fishbein  and  Ajzen,  1975)  suggests 
that  behaviors  can  lead  to  attitudes.  When  dissonance  with  a 
presently  held  attitude  is  created  by  the  performance  of  a 
contradictory  behavior,  the  individual  may  change  the  belief 
or  attitude  to  remove  or  reduce  the  dissonance.  To  justify 
continuing  use  of  the  system  (assuming  the  use  is  voluntary) , 
the  individuals  may  reevaluate  the  system  more  positively  to 
reduce  the  dissonance.  Dissonance  theory  supports  Figure  3.1 
which  suggests  that  system  usage  (a  behavior)  leads  to  user 
information  satisfaction  (an  attitude) .  In  the  case  where  use 
is  not  voluntary,  a  poor  quality  system  may  lead  to 
unfavorable  attitudes. 

B.  SYSTEM  USAGE  AND  PERFORMANCE 

Lucas  (1975)  states  that  evaluations  of  information 
systems  should  consider  the  relationship  between  the  use  of 
systems  and  performance.  He  says  that  in  some  cases,  use  of  a 
system  may  be  related  to  performance  and  in  other  instances 
performance  may  be  a  determinant  of  use.  He  discusses  three 
relationships  between  system  usage  and  performance.  The  first 
is  low  performance  predicts  high  levels  of  use  of  information 
provided  by  an  information  system.  This  applies  when  a  problem 
is  ill-defined  and  low  performance  stimulates  the  use  of 
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information,  such  as  comparison  with  past  performance.  For 
problem  solving,  the  decision  maker  considers  alternative 
actions,  evaluates  their  consequences,  selects  an  alternative 
and  implements  it. 

The  second  relationship  between  usage  and  performance  that 
Lucas  (1974)  describes  is  where  high  levels  of  usage  predict 
high  performance.  In  this  case  the  problem  is  well-defined  and 
an  information  system  providing  certain  information  (such  as 
market  forecasts)  can  contribute  directly  to  performance.  The 
strength  of  the  relationship  wil  depend  on  the  relevance  of 
the  information  and  the  problem  solver' s  ability  to  analyze 
alternatives  and  implement  the  selected  action.  Performance  is 
positively  related  to  usage. 

The  third  relationship  between  usage  and  performance 
occurs  when  the  information  provided  by  a  system  is  completely 
irrelevant.  Use  of  the  system  obscures  the  real  problems  and 
diverts  resources  which  could  be  better  allocated  to  other 
activities  (Lucas,  1974)  .  The  use  of  the  information  system  is 
dysfunctional:  under  these  conditions  less  usage  is  expected 
to  predict  higher  performance.  Usage  is  negatively  related  to 
performance . 

In  his  empirical  research  involving  a  sales  force  of  a 
manufacturing  firm,  Lucas  (1974)  found  that  usage  and 
performance  are  partially  determined  by  situational  factors 
such  as  length  of  time  in  a  position,  and  by  personal  factors 
such  as  age  and  education.  He  found  that  use  of  an  information 
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system  is  dependent  on  attitudes  and  perceptions  about  systems 
in  general.  Attitudes  have  an  action  component:  positive 
attitudes  toward  consistent  with  a  high  level  of  use  of  an 
information  system.  If  attitudes  are  highly  negative,  on  the 
other  hand,  users  will  minimize  the  contribution  of  the  system 
and  question  the  validity  of  outpu  reports.  He  reported  that 
attitudes  and  perceptions  of  computer  systems  are  influenced 
in  turn  by  the  quality  of  the  system  as  perceived  by  users. 

C.  SYSTEM  PERFORMANCE/  USEFULNESS  AFFECTS  PRODUCTIVITY 

When  an  information  system  is  performing  well  -  that  is, 
the  system  is  doing  what  it  is  supposed  to  do,  it  can 
positively  impact  the  productivity  of  the  persons  using  the 
system  as  well  that  of  the  environment  in  which  the  system 
operates  (Bearman,  et .  al.,  1985;  Schwartz,  1986).  Information 
is  an  essential  ingredient  in  office  work,  in  management 
decision  making  and  in  knowledge  worker  productivity. 
Automated  office  information  systems  are  designed  to  process 
information  more  efficiently  and  more  effectively  so  as  to 
increase  the  productivity  of  the  office  staff  and  the 
profitability  of  the  organization  (Borko, 1983) . 

Traditionally,  new  technology  was  used  to  facilitate 
productivity  gains  in  the  blue  collar  work  force  or  in  the 
clerical  population.  However,  in  recent  years,  there  has  been 
significant  recognition  that  productivity  gains  can  be 
realized  by  improving  the  technilogical  capabilities  of 
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knowledge  irforkers.  Knowledge  workers  include  the  managerial 
and  professional  population  (Bearman,  et .  al.,  1985;  Borko, 
1983)  .  In  order  to  increase  productivity  in  general,  it  is 
necessary  to  concentrate  on  increasing  the  productivity  of  the 
white  collar  worker,  and  one  way  by  which  this  can  be 
accomplished  is  by  an  increased  investment  in  office 
technology  (Borko,  1983). 

Mason  (1984)  noted  three  general  respects  in  which 
successful  performance  of  information  systems  can  increase 
product ivtiy .  The  first  is  by  reducing  costs;  another  is  by 
increasing  revenues  and  the  third  is  by  providing  better 
services,  Worthley  (1990)  said,  "To  hear  the  computer  vendors 
tell  it,  computer  technology  is  the  answer  to  a  broad  range  of 
productivity  needs  in  government" 

D.  PRODUCTIVITY  AND  COST  JUSTIFICATION 

Schwartz  (1986),  presented  a  new  methodology  for  cost 
justification  of  office  information  systems  in  engineering 
environments.  The  methodology  is  called  the  "hedonic"  wage 
model.  It  is  based  on  the  fact  that  the  allocation  of  labor 
resources  in  an  organization  tends  to  conform  to  certain 
logical  economic  criteria.  These  criteria,  along  with  the 
information  on  the  costs  of  labor  and  on  how  workers  spend 
their  time,  permit  inferences  about  the  organizational  values 
of  key  activities.  His  hedonic  wage  model  estimates  the 
implicit  marginal  values  to  the  organization  of  key  activities 
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by  using  data  on  organizatonal  structure,  work  activities, 
resource  allocation  and  costs.  These  marginal  values  provide 
the  basis  for  estimating  the  financial  impacts  of  a  proposed 
system  on  an  organization  by  placing  a  dollar  value  on 
improvements  in  worker  efficiency  and  effectiveness. 

Efficiency  refers  to  the  additional  amount  of  work  in  an 
activity  which  can  be  accomplished  in  the  same  amount  of  time. 
For  example,  if  an  information  system  allows  each  hour  spent 
in  data  analysis  to  produce  ten  percent  more  output,  the 
increase  in  efficiency  in  that  activity  is  ten  percent. 

Effectiveness  refers  to  the  amount  of  time  workers  spend 
doing  their  principal  activities  rather  than  support 
activities.  An  increase  in  effectiveness  is  achieved  when  an 
individual's  work  profile  is  modified  so  more  time  is  spent 
doing  high  value  rather  than  low  value  work.  The  work  profile 
is  a  matrix  identifying  how  much  time  is  routinely  spent  in 
each  of  the  key  acitvity  categories  by  each  type  of  worker.  In 
his  particular  study,  Schwartz  (986)  collected  data  for  a  work 
profile  using  a  time  log  which  was  distributed  to  a 
representative  sample  of  employees.  He  then  calculated  the 
implicit  marginal  values  of  the  key  activity  categories  in  the 
engineering  organization.  He  first  determined  whether  the 
number  of  employees  in  each  job  category  represented  an 
unconstrained  equilibrium  and  then  defined  and  solved  a  set  of 
simultaneous  equations  which  incorporated  the  data  and 
assumptions.  The  unknown  variables  in  the  equations  are  the 
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implicit  marginal  values  of  the  activity  categories  and  the 
coefficients  are  the  work  profile  perentages.  The  result  was 
a  more  credible  lower-bound  estimate  of  the  financial  value  of 
an  information  system  to  an  organization.  Lastly,  the  model 
used  the  cost-benefit  analysis.  This  produced  a  bottom-line 
value  for  the  proposed  information  system. 

E.  SUMMARY 

Linkages  exist  among  the  various  IS  effectiveness 
measures.  Some  of  the  linkages  which  are  prevalent  in  the  IS 
effectiveness  literature  are:  user  satisfaction  and  system 
usage;  system  usage  and  performance/usefulness;  performance 
and  productivity,  and  productivity  and  cost-benefit  analysis. 

Two  additional  linkages  which  emerge  in  the  IS 
effectiveness  literature  concern  user  involvement.  A 
discussion  of  this  takes  place  in  the  following  chapter. 
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IV.  USER  INVOLVEMENT:  A  RELATED  ISSUE 


A.  INTRODUCTION 

User  involvement  is  not  a  vehicle  for  measuring  the 
effectiveness  of  information  systems.  However,  it  is  an  issue 
that  emerges  in  the  IS  effectiveness  literature.  Its  effect  on 
the  surrogate  measures  user  satisfaction  and  system  usage 
(Figure  4.1)  warrant  brief  discussion  and  that  is  the  intent 
of  this  chapter. 

B .  USER  INVOLVEMENT 

Before  discussing  the  linkages  that  exist  between  user 
involvement  and  IS  effectiveness  measures,  it  is  appropriate 
to  discuss  user  involvement  as  a  separate  entity  related  to  IS 
effectiveness . 

When  considering  the  effectiveness  of  an  information 
system,  one  theme  is  becoming  more  prevalent  in  the  relevant 
literature.  That  is,  involving  the  users  in  the  conceptual 
phase  of  the  IS  and  soliciting  their  inputs  throughout  its 
implementation.  An  empirical  study  conducted  by  Baronas  and 
Louis  (1988)  concluded  that  users'  involvement  in  system 
installation  is  effective  because  it  restores  or  enhances 
their  perceived  control  over  their  work.  In  particular,  the 
study  involved  payroll  and  personnel  employees  from  35  state 
government  agencies.  Forty-three  people  were  introduced  to  a 
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Figure  4.1.  How  User  Involvement  Affects  User  Satisfaction 

and  System  Usage 


new  system  in  a  traditional  way  (no  formal  orientation) ,  while 
the  remaining  forty-nine  went  through  a  special  customized 
introduction  process  which  focused  on  making  them  feel 
comfortable  that  the  new  system  would  not  cause  them  to  lose 
control  over  their  work.  In  effect,  those  who  received  the 
special  training  were: 

1.  Significantly  more  satisfied  with  the  new  system 

2.  More  positive  in  their  perception  of  interacting 
with  systems  implementors 
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3. 


Happier  with  the  new  system  than  with  the  old 
systems . 


According  to  Ives  and  Olson  (1984),  user  involvement  is 
predicted  to  increase  user  acceptance  by: 


1.  Developing  realistic  expectations  about  system 
capabilities  (Gibson, 1977) ; 

2.  Providing  an  arena  for  bargaining  and  conflict 
resolution  about  design  issues  (Keen,  1981) ; 

3.  Leading  to  system  ownership  by  users  (Robey  and 
Farrow,  1982); 

4.  Decreasing  user  resistance  to  change  (Lucas, 
1974) ; 

5-  Commit ing  users  to  the  system  (Lucas, 1974;  Markus, 

1983)  . 


User  involvement  is  also  predicted  to  improve  system 
quality  by  (Ives  and  Olson,  1984)  : 


1.  Providing  a  more  accurate  and  complete  assessment 
of  use  information  requirements  (Norton  and 
McFarlan,  1975;  Robey  and  Farrow,  1982); 

2.  Providing  expertise  about  the  organization  the 
system  is  to  support  (Lucas,  1974); 

3.  Avoiding  development  of  unacceptable  or 
unimportant  features  (Robey  and  Farrow,  1982) ; 

4.  Improving  user  understanding  of  the  system  (Lucas, 
1974;  Robey  and  Farrow,  1982) . 


The  informations  systems  literature  shows  general 
agreement  that  the  success  of  information  systems  can  be 
improved  by  involving  the  user  in  the  development  of  those 
systems.  Involving  users  in  systems  development  and 
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implementation  can  impact  positively  on  other  aspects  of  IS 
effectiveness  such  as  system  usage  and  user  satisfaction. 

C.  USER  INVOLVEMENT  AFFECTS  SYSTEM  USAGE  AND 

USER  SATISFACTION 

User  involvement  in  information  system  development  is 
considered  an  important  mechanism  for  improving  system  quality 
and  ensuring  successful  implementation.  The  assumption  that 
user  involvement  leads  to  system  usage  and/or  user 
satisfaction  (Figure  4.1)  was  examined  in  an  empirical  study 
by  Baroudi,  et.al.  (1984)  in  a  survey  of  200  production 
managers.  The  results  demonstrated  that  user  involvement  in 
the  development  of  information  systems  will  enhance  both 
system  usage  and  the  user's  satisfaction  with  the  system. 

Baronas  and  Louis  (1988)  conducted  an  empirical  study 
involving  ninety-two  payroll  and  personnel  employees  from  35 
state  government  agencies  to  illustrate  the  point  that  user 
involvement  in  system  implementation  would  increase  overall 
satisfaction  with  the  information  system.  In  the  study,  43 
people  were  introduced  to  the  system  in  the  traditional  way, 
while  the  remaining  49  underwent  special  customized  training 
which  focused  on  making  them  feel  comfortable  that  the  new 
system  would  not  cause  them  to  lose  control  over  their  work. 
Those  who  received  the  special  training  were  significantly 
more  satisfied  with  the  new  system;  more  positive  in  their 
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perception  of  interacting  with  systems  implementors;  and 
happier  with  the  new  system  than  the  old  systems. 

Franz  and  Robey  (1986)  further  illustrate  the  impact  of 
user  involvement  in  an  empirical  study  investigating 
organizational  factors  related  to  user  involvement  in 
information  system  development  and  perceived  system 
usefulness.  They  used  a  sample  of  one  hundred  and  eighteen 
managers  in  thirty-four  companies.  Their  results  showed  that 
user  involvement  in  design  and  implementation  is  related 
positively  to  users'  perceptions  of  system  usefulness. 
Similarly,  Edstrom  (1977),  in  a  study  of  16  companies,  showed 
that  users  perceive  systems  to  be  more  successful  when  they 
are  able  to  influence  the  designs  through  effective 
communication . 

D.  TOO  MUCH  USER  INVOLVEMENT 

While  user  involvement  in  system  development  has  benefits. 
Cash,  et .  al.  point  out  that  the  complexities  of  developing  IS 
has  forced  the  creation  of  specialized  departments,  resulting 
in  a  series  of  strained  relationships  with  users  of  their 
service.  They  explain  that  this  has  been  an  enduring  headache 
from  the  start.  The  specialists  have  appropriately  developed 
their  own  language  systems  to  communicate  with  each  other 
while  general  management  has  quite  a  different  language. 
Coordination  issues  tend  to  be  complex. 
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According  to  Cash,  et .  al.  when  users  have  too  much 
dominance  in  system  development  the  following  implications 
emerge : 


1.  Too  much  emphasis  on  problem  focus 

2.  Information  technology  stays  out  of  control 

3.  Explosive  growth  in  the  number  of  new  systems  and 
supporting  staff 

4.  Multiple  suppliers  delivering  services 

5.  Lack  of  standardization  and  control  over  data 
hygiene  and  system 

6.  Hard  evidence  of  benefits  non-existent 

7.  Soft  evidence  of  benefits  not  organized 

8.  Few  measurements/objectives  for  new  systems 

9.  Technical  advice  of  IS  department  not  sought,  or 
if  received,  considered  irrelevant 

10.  User  buying  design/construction/maintainance 
services  and  even  operations  from  outside 

11.  User  building  networks  to  own  unique  needs  (not 
corporate  needs) 

12.  While  some  users  are  growing  rapidly  in  experience 
and  use,  other  users  feel  nothing  is  relevant 
because  they  do  not  understand 

13.  No  coordinated  effort  for  technology  transfer  or 
learning  from  experience  between  users 

14.  Growth  in  duplication  of  technical  staffs 

15.  Communications  costs  are  rising  dramatically 
through  redundancy 


A  user  tends  toward  short-term  need  fulfillment  (at  the 
expense  of  long-term  IS  hygiene  and  orderly  development) .  An 
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IS  department,  on  the  other  hand,  can  become  preoccupied  with 
the  mastery  of  technology  and  an  orderly  development  plan  at 
the  risk  of  slow  response  to  legimate  user  needs.  Effectively 
balancing  the  roles  of  these  two  groups  is  a  complex  task, 
which  must  be  dealt  with  in  the  context  of  the  corporate 
culture  and  the  potential  strategic  IS  role. 

E.  SUMMARY 

While  user  involvement  is  not  a  measure  of  IS 
effectiveness,  it  is  a  related  issue.  User  involvement  has 
been  shown  empirically  to  affect  user  satisfaction  and  system 
usage,  which  are  IS  effectiveness  measures.  Although  some 
researchers  advocate  involving  users  in  system  development  and 
implementation,  others  have  identified  potential  drawbacks  to 
too  much  user  involvement. 
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V.  CONCLUSION 


IS  effectiveness  is  concerned  with  those  effects  on  an 
organization  which  result  from  the  development  and  use  of  an 
information  system  (Carlson,  1974) .  It  can  also  be  described 
as  the  accomplishment  of  objectives  (Hamilton  and  Chervany, 
1981)  .  It  can  be  determined  by  comparing  performance  to 
objectives  or  by  attainment  of  a  normative  state  (Hamilton  and 
Chervany,  1981) .  Other  definitions  of  IS  effectiveness  are 
embedded  in  the  evaluation  techniques,  such  as  increased 
productivity,  a  satisfied  user  community,  or  successful 
performance  of  an  IS. 

The  numerous  and  various  approaches  to  measuring  IS 
effectiveness  can  be  classified  into  six  general  schools  of 
thought:  user  satisfaction,  system  usage, 
performance/usefulness,  productivity,  value  analysis  and  cost- 
benefit  analysis.  In  addition,  several  linkages  exist  among 
the  various  measures  of  IS  effectiveness.  These  include:  user 
satisfaction  and  system  usage;  system  usage  and  performance; 
performance  and  productivity,  and  productivity  and  cost- 
benefit  analysis. 

User  involvement,  while  not  an  IS  effectiveness  measure, 
is  a  related  issue  that  emerges  in  the  IS  effectiveness 
literature.  It  has  been  shown  to  affect  user  satisfaction  and 
system  usage.  While  there  are  many  advocates  of  user 
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involvement,  some  potential  drawbacks  to  too  much  user 
involvement  have  been  identified. 

The  definition  of  IS  effectiveness  remains  a  topic  of 
research  and  discussion  among  practitioners  and  researchers. 
Evaluation  of  IS  effectiveness  is  complicated  by  its  multi¬ 
dimensionality,  its  quantitative  and  qualitative  aspects,  and 
the  multiple  and  often  conflicting  evaluator  viewpoints.  For 
IS  effectiveness  researchers  and  practitioners,  it  will  be 
beneficial  to  incorporate  multiple  viewpoints  of  multiple 
objectives  and  performance  measures  into  the  complete 
assessment  of  IS  effectiveness  (Hamilton  and  Chervany,  1981) . 
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APPENDIX  A:  LIST  OF  JOURNALS 


COMMUNICATION  OF  THE  ACM 

COMPUTING  SURVEYS 

DATA  BASE 

DECISION  SCIENCES 

HUMAN  RELATIONS 

IEEE 

INFORMATION  AND  MANAGEMENT 
INFORMATION  PROCESSING  AND  MANAGEMENT 
INTERFACES 

JOURNAL  OF  SYSTEMS  MANAGEMENT 

JOURNAL  OF  THE  AMERICAN  SOCIETY  FOR  INFORMATION  SCIENCE 

JOURNAL  OF  THE  MANAGEMENT  INFORMATION  SYSTEMS 

MANAGEMENT  INFORMATICS 

MANAGEMENT  SCIENCE 

MIS  QUARTERLY 

OMEGA 

PUBLIC  PRODUCTIVITY  REVIEW 
SYSTEMS,  OBJECTIVES,  SOLUTIONS 
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APPENDIX  B:  BAILEY  AND  PEARSON'S  USER  SATISFACTION  SURVEY 


1.  Top  management  Involvement:  the  positive  or  negative 
degree  of  interest,  enthusiasm,  support,  or  participation  of 
any  management  level  above  the  user's  own  level  toward 
computer-based  information  systems  or  services  or  toward  the 
computer  staff  which  supports  them. 

strong  vs  weak 
consistent  vs  inconsistent 
good  vs  bad 

significant  vs  insignificant 

2.  Organizational  competition  with  the  IS  unit:  the 
contention  between  the  respondent's  organizational  unit  and 
the  IS  unit  when  vying  for  organizational  resources  or  for 
responsibility  for  success  or  failure  of  information  systems 
or  services  of  interest  to  both  parties. 

productive  vs  destructive 
rational  vs  emotional 
low  vs  high 

harmonious  vs  dissonant 

3.  Priorities  determination:  policies  and  procedures  which 
establish  precedence  for  the  allocation  of  IS  resources  and 
services  between  different  organizational  units  and  their 
requests . 


fair  vs  unfair 
consistent  vs  inconsistent 
just  vs  unjust 
precise  vs  vague 

4.  Charge-back  method  of  payment  for  services:  the  schedule 
of  charges  and  the  procedures  for  assessing  users  on  a  pro 
rata  basis  for  the  IS  resouces  and  services  that  they  utilize. 


just  vs 
reasonable  vs 
consistent  vs 
known  vs 


unjust 

unreasonable 

inconsistent 

unknown 
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5.  Relationship  with  the  IS  staff i  the  manner  and  methods  of 
interaction,  conduct,  and  association  between  the  user  and  the 
IS  staff. 


harmonious  vs  dissonant 
good  vs  bad 

cooperative  vs  uncooperative 
candid  vs  deceitful 

6.  Communication  with  the  IS  staff:  the  manner  and  methods  of 
information  exchange  between  the  user  and  IS  staff. 

harmonious  vs  dissonant 
productive  vs  destructive 
precise  vs  vague 
meaningful  vs  meaningless 

7.  Technical  competence  of  the  IS  staff:  the  computer 
technology  skills  and  expertise  exhibited  by  the  IS  staff. 

current  vs  obselete 
sufficient  vs  insufficient 
superior  vs  inferior 
high  vs  low 

8.  Attitude  of  the  IS  staff:  the  willingness  and  commitment 
of  the  IS  staff  to  subjugate  external  professional  goals  in 
favor  of  organizationally  directed  goals  and  tasks. 

user-oriented  vs  self-centered 
cooperative  vs  belligerent 
courteous  vs  discourteous 
positive  vs  negative 

9.  Schedule  of  products  and  services:  the  IS  center  timetable 
for  production  of  information  system  outputs  and  for  provision 
of  computer-based  services. 

good  vs  bad 
regular  vs  irregular 
reasonable  vs  unreasonable 
acceptable  vs  unacceptable 
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10.  Time  required  for  new  development:  the  elapsed  time 
between  the  user's  request  for  new  applications  and  the 
design,  development,  and  implementation  of  the  application 
systems  by  the  IS  staff. 


short  vs  long 

dependable  vs  undependable 
reasonable  vs  unreasonable 
acceptable  vs  unacceptable 

11.  Processing  of  change  recfuests:  the  manner,  method  and 
required  time  with  which  the  IS  staff  responds  to  user 
requests  for  changes  in  existing  computer-based  information 
systems  or  services. 


fast  vs  slow 
timely  vs  untimely 
simple  vs  complex 
flexible  vs  rigid 

12.  Vendor  support:  the  type  and  quality  of  the  ser  ice 
rendered  by  a  vendor,  either  directly  or  indirectly,  to  the 
user  to  maintain  the  hardware  or  software  required  by  that 
organizational  status. 

skilled  vs  bungling 
sufficient  vs  insufficient 
eager  vs  indifferent 
consistent  vs  inconsistent 

13.  Response/turnaround  time:  the  elapsed  time  between  a 
user-initiated  reques  for  service  and  a  reply  to  that  request. 
Response  time  generally  refers  to  the  elapsed  time  for 
terminal  type  request  or  entry.  Turnaround  time  generally 
refers  to  the  elapsed  time  for  execution  of  a  program 
submitted  or  requested  by  a  user  and  the  return  of  the  output 
to  that  user. 


fast  vs  slow 
good  vs  bad 

consistent  vs  inconsistent 
reasonable  vs  unreasonable 


14.  Means  of  input /output :  the  method  and  medium  by  which  a 
user  inputs  data  to  and  receives  output  from  the  IS  center. 


convenient  vs  inconvenient 
clear  vs  hazy 
efficient  vs  inefficient 
organized  vs  disorganized 
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15.  Convenience  of  access:  the  ease  or  difficulty  with  which 
the  user  may  act  to  utilize  the  capability  of  the  computer 
system. 

convenient  vs  inconvenient 
good  vs  bad 
easy  vs  difficult 
efficient  vs  inefficient 

16.  Accuracy:  the  correctness  of  the  output  information. 

accurate  vs  inaccurate 
high  vs  low 

consistent  vs  inconsistent 
sufficient  vs  insufficient 

17.  Timeliness:  the  availability  of  the  output  information  at 
a  time  suitable  for  its  use. 

timely  vs  untimely 
reasonable  vs  unreasonable 
consistent  vs  inconsistent 
punctual  vs  tardy 


18.  Precision:  the  variability  of  the  output  information  form 
that  which  it  purports  to  measure. 

sufficient  vs  insufficient 
consistent  vs  inconsistent 
high  vs  low 
definite  vs  uncertain 

19.  Reliability:  the  consistency  and  dependability  of  the 
output  information. 

consistent  vs  inconsistent 
high  vs  low 
superior  vs  inferior 
sufficient  vs  insufficient 

20.  Currency:  the  age  of  the  output  information. 

good  vs  bad 
timely  vs  untimely 
adequate  vs  inadequate 
reasonable  vs  unreasonable 
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21.  Completeness:  the  comprehensiveness  of  the  output 
information  content. 

complete  vs  incomplete 
consistent  vs  inconsistent 
sufficient  vs  insufficient 
adequate  vs  inadequate 

22.  Format  of  output:  the  material  design  of  the  layout  and 
display  of  the  output  contents. 

good  vs  bad 
simple  vs  complex 
readable  vs  unreadable 
useful  vs  useless 

23.  Language:  the  set  of  vocabulary,  syntax,  and  grammatical 
rules  used  to  interact  with  the  computer  systems. 

simple  vs  complex 
powerful  vs  weak 
easy  vs  difficult 
easy-to-use  vs  hard-to-use 

24.  Volume  of  output:  the  amount  of  information  conveyed  to 
a  user  from  computer-based  systems.  This  is  expressed  not  only 
by  the  number  of  reports  or  outputs  but  also  by  the 
voluminousness  of  the  output  contents. 

concise  vs  redundant 
sufficient  vs  insufficient 
necessary  vs  unnecessary 
reasonable  vs  unreasonable 

25.  Relevancy:  the  degree  of  confruence  between  what  the  user 
wants  or  requires  and  what  is  provided  by  the  information 
products  and  services. 


useful  vs  useless 
relevant  vs  irrelevant 
clear  vs  hazy 
good  vs  bad 

26.  Error  recovery:  the  methods  and  policies  governing 
correction  and  rerun  of  system  outputs  that  incorrect. 


fast  vs  slow 
superior  vs  inferior 
complete  vs  incomplete 
simple  vs  complex 
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27.  Security  of  data:  the  safeguarding  of  data  from 
misappropriation  or  unauthorized  alterationor  loss. 


secure  vs  insecure 
good  vs  bad 
definite  vs  uncertain 
complete  vs  incomplete 


28.  Documentation:  the  recorded  description  of  an  information 
system.  This  includes  formal  instructions  for  the  utilization 
of  the  system. 

clear  vs  hazy 
available  vs  unavailable 
complete  vs  incomplete 
current  vs  obselete 

29.  Expectations:  the  set  of  attributes  or  features  of  the 
computer-based  informtion  products  or  services  that  a  user 
considers  reasonable  and  due  from  the  computer-based 
information  support  rendered  within  his  organization. 

pleased  vs  displeased 
high  vs  low 
definite  vs  uncertain 
optimistic  vs  pessimistic 

30.  Understanding  of  systems:  the  degree  of  comprehension 
that  a  user  possesses  about  the  computer-based  information 
systems  or  services  that  are  provided. 

high  vs  low 

sufficient  vs  insufficient 
complete  vs  incomplete 
easy  vs  hard 

31.  Perceived  utility:  the  user's  judgement  about  the 
relative  balance  between  the  cost  and  tbe  considered 
usefulness  of  the  computer-based  informtion  products  or 
services  that  are  provided.  The  costs  include  any  costs 
related  to  providing  the  resource,  including  money,  time, 
manpower,  and  opportunity.  The  usefulness  includes  any 
benefits  that  the  user  believes  to  be  derived  from  the 
support . 


high  vs  low 
positive  vs  negative 
sufficient  vs  insufficient 
useful  vs  useless 
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32.  Confidence  in  the  systems:  the  user's  feelings  of 
assurance  or  certainty  about  the  systems  provided. 

high  vs  low 
strong  vs  weak 
definite  vs  uncertain 
good  vs  bad 

33.  Feeling  of  participation:  the  degree  of  involvement  and 
commitment  which  the  user  shares  with  the  IS  staff  and  others 
toward  the  functioning  of  the  computer-based  information 
systems  and  services. 

positive  vs  negative 
encouraged  vs  repelled 
sufficient  vs  insufficient 
involved  vs  uninvolved 

34.  Feeling  of  control:  the  user's  awareness  of  the  personal 
power  or  lack  of  power  to  regulate,  direct  or  dominate  the 
development,  alteration,  and  the  execution  of  the  computer- 
based  information  systems  or  services  which  serve  the  user's 
perceived  function. 


high  vs  low 

sufficient  vs  insufficient 
precise  vs  vague 
strong  vs  weak 

35.  Degree  of  training:  the  amount  of  specialized  instruction 
and  practice  that  is  afforded  to  the  user  to  increase  the 
user's  proficiency  in  utilizing  the  computer  capability  that 
is  available. 


complete  vs  incomplete 
sufficient  vs  insufficient 
high  vs  low 
superior  vs  inferior 

36.  Job  effects:  the  changes  in  job  freedom  and  job 
performance  that  are  ascertained  by  the  user  as  resulting  from 
modifications  induced  by  the  computer-based  information 
systems  and  services. 

liberating  vs  inhibiting 
significant  vs  insignificant 
good  vs  bad 
valuable  vs  worthless 
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37.  Organizational  Positions  of  the  IS  function:  the 
hierarchical  relationship  of  the  IS  function  to  the  overall 
organizational  structure. 

appropriate  vs  inappropriate 
strong  vs  weak 
clear  vs  hazy 
progressive  vs  regressive 

38.  Flexibility  of  Systems:  the  capacity  of  the  information 
system  to  change  or  to  adjust  in  response  to  new  conditions, 
demands,  or  circumstances. 

flexible  vs  rigid 
versatile  vs  limited 
sufficient  vs  insufficient 
high  vs  low 

39.  Integration  of  Systems:  the  ability  of  systems  to 
communicate/transmit  data  between  systems  servicing  different 
functional  areas. 


complete  vs  incomplete 
sufficient  vs  insufficient 
successful  vs  unsuccessful 
good  vs  bad 
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APPENDIX  C:  "DETAILED"  USER  SATISFACTION  INSTRUMENT 
Ives,  Olson  and  Baroudi  (1983)  Survey 

This  survey  is  designed  to  measure  your  personal  feelings 
about  the  computer-based  informtion  systems  used  at  your  firm. 

Please  check  each  scale  in  the  position  that  describes 
your  evaluation  of  the  factor  being  judged.  Please  check  only 
one  position  on  each  scale  in  (rather  than  between)  the  space 
provided. 

Work  rapidly;  rely  on  your  first  impressions.  Please  do 
not  omit  any  scale. 

1.  Relationship  with  IS  staff: 

harmonious  : _ : _ : _ : _ : _ : _ : _ :  dissonant 

good  : _ : _ : _ : _ : _ : _ : _ :  bad 

2.  Processing  of  requests  for  changes  to  existing  systems: 

fast  : _ : _ : _ : _ : _ : _ : _ :  slow 

timely  : _ : _ : _ ; _ : _ : _ : _ :  untimely 

3.  Degree  of  IS  training  provided  to  users: 

complete  : _ : _ : _ : _ :  : _ : _ :  incomplete 

high  : _ : _ : _ : _ : _ : _ : _ :  low 

4.  Users'  understanding  of  systems: 

sufficient  : _ : _ : _ : _ : _ : _ : _ :  insufficient 

complete  ::::::::  incomplete 
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5. 


Users'  feelings  of  participation: 


positive  ; _ : _ : _ : _ : _ : _ : _ :  negative 

sufficient  : _ : _ : _ : _ : _ : _ : _ :  insufficient 

6.  Attitude  of  the  IS  staff: 

cooperative  : _ : _ : _ : _ : _ : _ : _ :  belligerent 

positive  : _ : _ : _ : _ : _ : _ : _ :  negative 

7.  Reliability  of  output  information: 

high  : _ : _ : _ : _ : _ : _ : _ :  low 

superior  : _ : _ : _ : _ : _ : _ : _ :  inferior 

8.  Relevancy  of  output  information  (to  intended  function) 

useful  : _ : _ : _ : _ : _ : _ : _ :  useless 

relevant  : _ : _ : _ : _ : _ : _ : _ :  irrelevant 

9.  Accuracy  of  output  information: 

accurate  : _ : _ : _ : _ : _ : _ : _ :  inaccurate 

high  : _ : _ : _ : _ : _ : _ : _ :  low 

10.  Precision  of  output  information: 

high  : _ : _ : _ : _ : _ : _ : _ :  low 

definite  :  :_  : _ : _ : _ :  :  _:  indefinite 

11.  Communication  with  the  IS  staff: 

harmonious  : _ : _ : _ : _ : _ : _ : _ :  dissonant 

productive  : _ : _ : _ : _ : _ : _ : _ :  unproductive 

12.  Time  required  for  new  systems  development: 

reasonable  : _ : _ : _ : _ : _ : _ : _ :  unreasonable 

acceptable  :  :  :  :  :  : _ : _ :  unacceptable 
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13.  Completeness  of  the  output  information: 

sufficient  : _ : _ : _ : _ : _ : _ : _ :  insufficient 

adequate  :  :  :  :  : _ : _ : _ :  inadequate 


Summary 

1.  How  satisfied  are  you  with  your  involvement  and 
participation  in  the  operation  and  ongoing  development  of 
information  systems? 

satisfied  ::::::::  not  satisfied 


2.  How  satisfied  are  you  with  the  support  and  services  of  the 
IS  department? 

satisfied  ::::::::  not  satisfied 


3.  How  satisfied  are  you  with  the  support  and  services  of  the 
IS  department? 

satisfied  : _ : _ : _ : _ : _ : _ : _ :  not  satisfied 

4.  In  summary,  how  satisfied  are  you  with  the  entire 
information  systems  environment? 

satisfied  not  satisfied 


Are  you  primarily  an  information  system  user  or  developer? 
Please  check  only  one.  _ user  _ developer 
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